Efficient Transformers: A Survey

Transformer model architectures have garnered immense interest lately due to their effectiveness across a range of domains like language, vision, and reinforcement learning. In the field natural language processing for example, Transformers become an indispensable staple in modern deep learning stack. Recently, dizzying number “X-former” models been proposed—Reformer, Linformer, Performer, Longformer, name few—which improve upon original architecture, many which make improvements around computational memory efficiency . With aim helping avid researcher navigate this flurry, article characterizes large thoughtful selection recent efficiency-flavored models, providing organized comprehensive overview existing work multiple domains.